Method and Generator for Generating Disturbed Input Data for a Neural Network

ABSTRACT

The invention relates to a method for generating disturbed input data for a neural network for analyzing sensor data, in particular digital images, of a driver assistance system, in which a first metric is defined which indicates how the magnitude of a change in sensor data is measured, a second metric is defined which indicates where a disturbance of sensor data is directed, an optimization problem is generated from a combination of the first metric and second metric, the optimization problem is solved by means of at least one solution algorithm, wherein the solution indicates a target disturbance of the input data, and disturbed input data is generated from sensor data for the neural network by means of the target disturbance.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to German Patent Application No. DE 102019 208 733.7, filed on Jun. 14, 2019 with the German Patent andTrademark Office. The contents of the aforesaid Patent Application areincorporated herein for all purposes.

TECHNICAL FIELD

The present invention relates to a method for generating disturbed inputdata for a neural network for analyzing sensor data, in particular foranalyzing digital images, of a driver assistance system. The inventionfurther relates to a method for checking the robustness of such a neuralnetwork and to a method for improving a parameter set of such a neuralnetwork. Moreover, the invention relates to a generator for generatingdisturbed input data for a neural network for analyzing sensor data, inparticular for analyzing digital images, of a driver assistance system.

BACKGROUND

This background section is provided for the purpose of generallydescribing the context of the disclosure. Work of the presently namedinventor(s), to the extent the work is described in this backgroundsection, as well as aspects of the description that may not otherwisequalify as prior art at the time of filing, are neither expressly norimpliedly admitted as prior art against the present disclosure.

Modern vehicles comprise driver assistance systems which support thedriver in the controlling of the vehicle or partially or completelytakes over the task of driving. By using such driver assistance systems,various levels of vehicle control automation may be achieved. At a lowlevel of automation, only information and warnings are output to thedriver. At higher levels of automation, the driver assistance systemactively intervenes in the control of the vehicle. For example, thesteering of the vehicle or the acceleration in the positive or negativedirection is intervened with. In the case of an even higher level ofautomation, apparatuses of the vehicle are intervened with to such anextent that certain locomotion types of the vehicle, for examplestraight-ahead driving, may be executed automatically. At the highestlevel of automation, the vehicle may drive autonomously.

In such driver assistance systems, the analysis of digital images thatare taken in the surroundings of the vehicle during the drive are ofessential importance. The driver assistance system may only safelycontrol the vehicle when the digital images are correctly analyzed.Machine learning has great potential in the analysis of digital imagesof a driver assistance system. The raw sensor data, which is generated,for example, by a camera, a radar sensor, or a lidar sensor of avehicle, is processed by means of a deep neural network. The neuralnetwork generates output data, from which the driver assistance systemderives relevant information for partially automated or fully automateddriving. For example, the type and position of objects in the vicinityof the vehicle and their behavior are ascertained. Furthermore, theroadway geometry and roadway topology may be ascertained by means ofneural networks. Convolutional neural networks are particularly suitablefor processing digital images.

Such deep neural networks are trained for use in a driver assistancesystem. In this case, the parameters of the neural network may besuitably adapted through the input of data without a human expertneeding to intervene. For a given parameterization, the deviation of anoutput of a neural network from a ground truth is measured. Thisdeviation is also described as “loss.” Here, what is known as a lossfunction is chosen in such a way that the parameters depend on it in amanner that may be differentiated. Using a gradient descent, in eachtraining step the parameters of the neural network are then adapteddepending on the derivation of the deviation, which is ascertained onthe basis of multiple examples. These training steps are repeated veryoften until the deviation, i.e., the loss, no longer decreases.

In this approach, the parameters are ascertained without the assessmentof a human expert or a semantically motivated model. For the neuralnetworks, this means that they are often largely nontransparent forhumans and their calculations cannot be interpreted. This leads to inparticular deep neural networks often not being able to besystematically tested or formally verified.

Moreover, this results in the problem that deep neural networks aresusceptible to harmful disruptive influences (adversarialperturbations). Small manipulations to the input data that are barelyperceptible or not perceptible at all to humans or manipulations that donot change the situational assessment may lead to output data thatdiffers considerably from the output data that would have resultedwithout the manipulation. Such manipulations may be either changes tothe sensor data caused maliciously or randomly occurring image changesdue to sensor noise, weather influences, or certain colors andcontrasts.

It cannot be predicted to which input features a neural network reactsso sensitively that the output data changes considerably even in thecase of small changes to the input data. This means that synthetic datacannot be used successfully for training neural networks used in suchdriver assistance systems. It has been shown that neural networks thathave been trained in simulations or on otherwise synthetic data performpoorly when used in a driver assistance system with real sensor data. Ithas also been shown that executing a driver assistance system with aneural network in a different domain may also starkly reduce thefunctional quality. For example, it may occur that a driver assistancesystem with a neural network that was trained in the summer isunsuitable for execution in the winter. The development and approval ofneural networks for driver assistance systems on the basis of asimulation is therefore problematic.

There is therefore a need to develop neural networks for driverassistance systems which are robust against disturbances. The neuralnetworks should then also generate usable output data for the driverassistance system when the input data is disturbed.

To achieve this, it is known to generate disturbed input data for aneural network by means of known disturbances and test how the outputdata of the neural network reacts to this disturbed input data. Thereare collections for disturbances of the input data, by means of which itmay be tested how robust a neural network is against such disturbances.However, this results in the problem that disturbed input data may onlybe generated to a limited extent by known disturbances. There istherefore a need to generate disturbed input data for a neural networkfor analyzing sensor data, in particular digital images, of a driverassistance system in order to test and improve neural networks.

SUMMARY

A need exists to provide a method and a generator for generatingdisturbed input data for a neural network for analyzing sensor data of adriver assistance system, with which new disturbed input data for theneural network may be generated in a simple manner.

The need is addressed by methods and generators having the features ofthe independent claims. Embodiments of the invention are described inthe dependent claims, the following description, and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically shows an exemplary embodiment of a generator;

FIG. 2 schematically shows an exemplary embodiment of a method forgenerating disturbed input data;

FIG. 3 schematically shows an exemplary embodiment of a device forgenerating a parameter set;

FIG. 4 schematically shows an exemplary embodiment of a method forchecking the robustness of a neural network;

FIG. 5 schematically shows an exemplary embodiment of a method forimproving a parameter set of a neural network; and

FIGS. 6A-6C show an example of a disturbance.

DESCRIPTION

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features will be apparent fromthe description, drawings, and from the claims.

In the following description of embodiments of the invention, specificdetails are described in order to provide a thorough understanding ofthe invention. However, it will be apparent to one of ordinary skill inthe art that the invention may be practiced without these specificdetails. In other instances, well-known features have not been describedin detail to avoid unnecessarily complicating the instant description.

In a method according to a first exemplary aspect for generatingdisturbed input data for a neural network for analyzing sensor data of adrive assistance system, a first metric is defined which indicates howthe magnitude of a change of a digital image is measured, and a secondmetric is defined which indicates where a disturbance of the input datais directed. An optimization problem is generated from a combination ofthe first metric and the second metric. The optimization problem issolved by means of at least one solution algorithm, wherein the solutionindicates a target disturbance of the input data, and input datadisturbed by means of the target disturbance is generated from sensordata for the neural network.

The sensor data is in particular digital images. In this case, thetarget disturbance thus generates disturbed, i.e., changed digitalimages, which form the input data for the neural network that analyzesthe digital image.

In the method according to this exemplary aspect, possible harmfuldisturbances of a neural network which is used to analyze sensor dataare considered at a structural level. The disturbance is considered as acomposition of various elements for which various metrics are defined.Surprisingly, it was achieved as a result that no longer only randomlycomposed disturbances are used, but rather it is possible to generate alarge number of new harmful target disturbances on the basis of knowndisturbances by analyzing the structure of known disturbances withregard to the metrics.

In the method according to this exemplary aspect, an optimizationproblem is generated from two metrics which measure changes in sensordata, in particular in a digital image. For such an optimizationproblem, there are a large number of known solution algorithms. Theoptimization problem may thus be solved with these solution algorithms.As a result, a target disturbance of the input data is generated. Bymeans of this target disturbance, disturbed input data may then begenerated from sensor data for the neural network. The neural networkmay then be tested and trained on the basis of this disturbed inputdata. The method enables new disturbances to be generated very quicklyand simply.

The first metric used in the method according to this exemplary aspectindicates how the magnitude of a change in sensor data is measured. Ifthe sensor data is a digital image from a camera, the disturbance shouldusually be as small as possible to test the neural network. The firstmetric indicates how the magnitude of the change in the digital imagemay be quantified. A digital image may, for example, be changed in thatthe pixels of the image are shifted, rotated, or mirrored. The firstmetric indicates the magnitude of the change in the case of suchtransformations. According to the first metric, a rotation ortranslation of a digital image may be defined by a fixed point, and therotation angle or the translation distance in the horizontal andvertical direction, respectively. Furthermore, for each pixel in theimage, the first metric may determine the image distances in that thesum of the differences of all pixel values is ascertained. The pixelvalue may be a gray scale value and or a color value, for example. Foreach pixel, the difference of the pixel value for the original image andfor the disturbed image is formed. This difference is determined foreach pixel and the differences are then added. The result is an imagedistance which indicates the difference of the two images according tothe first metric.

Moreover, according to the first metric, changed image regions may beconsidered. The image regions may be defined by a starting point and anextension in the horizontal and vertical directions, or by a list ofpixels. According to the first metric, image distances may be determinedfor these image regions.

Moreover, the first metric may indicate the magnitude of a change in adigital image with reference to image characteristics, such asluminance, contrast, and/or structure values, or any combinationsthereof.

The definition of the first metric may also contain limitations, forexample that the changes that are considered in the first metric takeinto account only such image regions in which, for example, specificimage characteristics are present. For example, only such regions inwhich the contrast exceeds a specific threshold may be considered.

In some embodiments, the first metric is selected from first metricsthat measure potential naturally occurring disturbances, becausedisturbances ascertained by these metrics may actually occur duringexecution in the field. Such natural disturbances are, for example,changes in the sensor data that are generated due to weather influences,such as fog or snow, sensor noise, or by a dirty camera, or that aregenerated by textures.

Furthermore, naturally occurring disturbances are naturally occurringobjects in the surroundings of a vehicle, such as printed posters orstickers on objects. If, for example, the disturbance of the secondmetric is directed at making objects of a specific class disappear, itis possible to add a printed poster, a sticker on an object, fog, ortextures to a digital image. By means of such disturbances, whichaccording to the second metric is directed at a specific effect in thesensor data, such disturbed input data may be generated for a neuralnetwork, which disturbed input data is particularly relevant for the usein a driver assistance system.

In some embodiments, the second metric is directed at a change in theclassification of objects. It measures in particular the deviation ofthe true model output from the desired false model output, meaning thetarget of the adversarial disturbance. In a digital image, for example,small image regions or a small number of pixels may be disturbed so thatan object in the digital image is no longer detected as a trafficparticipant, such as a pedestrian, but instead as another classifiedobject, for example a region of a roadway. Furthermore, the disturbancemay be directed at, whenever a region is detected as a road, this roadalways being detected as an empty road without other trafficparticipants.

In some embodiments, the second metric may be directed at adisappearance of objects. The disturbance is, for example, directed atchanging detected objects so that they disappear. The second metric mayalso relate here only to specific image regions. For example, thedisturbance described by the second metric may be directed at objects ofa specific class not being able to be present in a specific imageregion.

In some embodiments of the method, the second metric is directed at achange in an object of a specific class. For example, an object may bedetected and classified. For example, an image region may be assigned toa traffic participant. The second metric is then, for example, directedat showing this object larger or smaller or at a different position. Forexample, objects classified as pedestrians may be shown smaller orlarger. In this case, the enlargement is defined, for example, by theabsolute indication of pixels through which the object is enlarged orshrunk to the left, right, top, and bottom by the disturbance.

There are a large number of possible disturbances that may be describedby the second metric. Any changes to the sensor data could be caused tochange the sensor data such that, in the analysis of the sensor data ina driver assistance system, results relevant to safety in particular maylonger be achieved correctly. For example, a pattern or a raster may beapplied to the sensor data so that objects of a specific class, forexample, pedestrians, in a digital image disappear but other objectscontinue to be correctly classified. For the application of the methodaccording to the present exemplary aspect in a driver assistance system,in particular such second metrics are relevant that measure thenaturally appearing disturbances: the model output appears plausible butdeviates from the truth in specific, safety-relevant details.

In some embodiments, the disturbances that are described by the firstand/or second metric are naturally occurring disturbances. For theapplication in a driver assistance system, a selection is thus made forthe possible disturbances that are described by the first and/or secondmetric that are particularly relevant for checking and improving neuralnetworks for use in a driver assistance system.

In some embodiments, the first and/or second metrics are stored in adatabase. A data set on a naturally occurring disturbance that ismeasured with the first and/or second metric is then loaded from thedatabase. The metrics for possible disturbances to the input data (firstmetrics) and for possible changes to the model outputs (second metrics)may be stored, for example, in the database. According to someembodiments, a data set on a naturally occurring disturbance (measuredwith a first metric) and for a possible target (a notified change in themodel output—e.g., overlooking all pedestrians—measured with a secondmetric) is then loaded from the database.

In some embodiments, a third metric is defined, which indicates whatkind of sensor data a third disturbance is applied to. For example, ifthe disturbance is applied to all data, to only one data point, or todata with specific conditions, for example for all data with multi-laneroads. The optimization problem is then generated from a combination ofat least two metrics of the first, the second, and the third metric. Theoptimization problem is in particular generated from a combination ofthe first, the second, and the third metric. The sensor data is inparticular digital images. These are analyzed in particular by a neuralnetwork in a driver assistance system.

The third metric may relate in particular to all sensor data, forexample all digital images. For example, the disturbance may result, inall digital images, in objects of a specific class disappearing.

Furthermore, the third metric may have an impact only on a subset of thesensor data, in particular of the digital images. The disturbance maydescribe, for example, only such digital images that contain objects ofa specific class, for example objects classified as pedestrians.Furthermore, the third metric may describe digital images that weretaken on days with snowfall or rain. As a result, the disturbed inputdata for the neural network in a use in a driver assistance system mayeffect, for example, a different evaluation of a special trafficsituation or environmental situation.

In some embodiments, the third metric describes only sensor data thatcontain a specific object. Alternatively or additionally, the thirdmetric may select only one specific digital image.

The optimization problem which has been generated on the basis of themetrics may be shown, for example, as follows: In a specified maximumchange in a digital image, for example, by rotating a specific imageregion, the number of pixels classified as persons should be minimized,for as many images as possible in which persons are present.

In another example, in a minimum change in the starting image in regionswith high contrast, the number of pixels classified as persons should beminimized, for as many images as possible in which persons are present.

For such optimization problems, a solution algorithm is indicated in themethod according to this exemplary aspect. In some embodiments, thesolution algorithm comprises iterative methods using the gradients ofthe neural network for determining the change directions. Furthermore,iterative methods using sampling, evaluation, and combinations thereofare used.

In some embodiments, a Monte Carlo method is used as a solutionalgorithm, in which, for example, a noise is generated for a digitalimage, and the result is checked. In some embodiments, a geneticalgorithm may be used for solving the optimization problem.

The solution to the optimization problem may be, for example, adisturbed digital image or a disturbance with which sensor data may bedisturbed to generate disturbed input data for a neural network. Thedisturbed sensor data or the disturbed digital image are then the inputdata for the neural network which is to be checked. A disturbance mayalso be applied to a set of input data by combination at the pixellevel, for example by summation.

Another exemplary aspect relates to a method for generating disturbedinput data for a neural network for analyzing sensor data, in particulardigital images, of a driver assistance system, in which a first quantityis defined that contains the first metrics, which each differentlyindicate how the magnitude of a change in sensor data is measured, asecond quantity is defined that contains second metrics, which eachdifferently indicate where a disturbance of sensor data is directed, anycombination of a first metric of the first quantity and a second metricof the second quantity is chosen, an optimization problem is generatedfrom the chosen combination of the first and second metric, theoptimization problem is solved by means of at least one solutionalgorithm, wherein the solution indicates a target disturbance of theinput data, and disturbed input data are generated by means of thetarget disturbance from sensor data for the neural network.

A benefit of this method is that any first metric of the first quantityand any second metric of the second quantity may be used to get to atarget disturbance by solving the optimization problem. The more metricsthe first and second quantities contain, the more different targetdisturbances may be generated by the method. A large number of targetdisturbances may thus be generated.

In some embodiments, the first quantity comprises at least two, inparticular at least five, different first metrics. However, the firstmetric may also contain more than 10, 20, or more than 100 metrics.

According to some embodiments, the second quantity comprises at leasttwo, in particular at least five, different second metrics. However, thesecond metric may also contain more than 10, 20, or more than 100metrics.

The first and/or the second metric of the first and/or second quantity,respectively, may in particular have the features as described above,individually or in combination.

In some embodiments, a third metric is defined, which indicates whatkind of sensor data a disturbance is applied to, and any combination ofa first metric of the first quantity, a second metric of the secondquantity, and the third metric is chosen. From the chosen combination ofthe first, second, and third metric, an optimization problem is thengenerated.

The third metric may have in particular the features as described above,individually or in combination.

In some embodiments, a solution algorithm quantity is defined thatcontains multiple solution algorithms that each solve the optimizationproblem differently in order to generate different target disturbancesof the input data. Any one solution algorithm of the solution algorithmquantity is then selected in order to generate disturbed input data fromsensor data for the neural network. In this manner, an even largernumber of target disturbances may be generated, because the solutionalgorithm may also be varied, wherein each solution algorithm comes todifferent target disturbances.

The solution algorithms of the solution algorithm quantity may compriseiterative methods using the gradients of the neural network fordetermining the change directions as well as sampling-based methods,gradient-based methods with momentum, and/or surrogate model-basedmethods.

The teachings herein also relate to a method for checking the robustnessof a neural network for analyzing sensor data, in particular digitalimages, against disturbed input data, in which the following steps areperformed: providing a neural network having an associated parameterset, generating training data by means of an example sensor data set,generating a first analysis of the example sensor data set on the basisof the training data by means of the neural network, generatingdisturbed input data as training data for the example sensor data set bymeans of the method described in the preceding for generating disturbedinput data for a neural network, generating a second analysis of theexample sensor data set on the basis of the disturbed input data bymeans of the neural network, comparing the first and second analysis,and ascertaining a robustness value depending on the result of thecomparison of the first and second analysis.

The teachings herein also relate to a method for improving a parameterset of a neural network for analyzing sensor data, in particular digitalimages, in relation to disturbed input data. With the method, thefollowing steps are performed:

-   -   a. providing a neural network having the associated parameter        set,    -   b. generating training data by means of an example sensor data        set,    -   c. generating a first analysis of the example data set on the        basis of the training data by means of the neural network,    -   d. generating disturbed input data as training data for the        example sensor data set by means of the method described in the        preceding for generating disturbed input data for a neural        network,    -   e. generating a second analysis of the example sensor data set        on the basis of the disturbed input data by means of the neural        network,    -   f. comparing the first and second analysis, and    -   g. ascertaining a robustness value depending on the result of        the comparison of the first and second analysis.

The example sensor data set is in particular a digital example image.

By means of the method, the method described at the outset forgenerating disturbed input data may be used to check how robust a neuralnetwork for analyzing sensor data is against the disturbed input data.If the neural network is used in a method for analyzing sensor data fora driver assistance system, it is important for safety when operatingthe vehicle which the driver assistance system is affecting that theneural network is robust against disturbed input data for the neuralnetwork. The neural network is robust against such disturbed input datawhen the deviation in the first and the second analysis is very low. Thedisturbed input data then has a low influence on the output of theneural network. However, when the disturbed input data lead to a verylarge deviation of the second analysis of the example sensor data setfrom the first analysis, even if the disturbances to the sensor data areonly very small, the neural network is not robust against thedisturbance of the input data.

If the sensor data is digital images, the first and second analysis maycomprise semantic segmentation of the digital image, detecting objectsin the digital image, classifying objects in the digital image, ordetecting the position of an object in the digital image. In addition,it may be detected by means of the analyses how an object in the digitalimage changes. These analyses are particularly relevant in a use of theneural network in a driver assistance system, so that it is importantthat the neural network is robust against disturbances that may occur insuch analyses, and thus that small changes occur in the analysis whendisturbed input data is used.

The teachings herein also relate to a method for improving a parameterset of a neural network for analyzing sensor data, in particular digitalimages, in relation to disturbed input data. The method comprises thesteps a. to f., as indicated in the preceding. Then, in a step h., animproved parameter set for the neural network is generated on the basisof the result of the comparison of the first and second analysis.

The improved parameter set is obtained by training the neural network.The training is performed for disturbed and undisturbed sensor data,i.e., in particular digital images. The improved parameter set thenresults, for example, from a gradient descent (adversarial training).

Moreover, the teachings herein relate to a generator for generatingdisturbed input data for a neural network for analyzing sensor data, inparticular digital images, of a driver assistance system with a firstmetric unit with a first metric, which indicates how the magnitude of achange in sensor data is measured, a second metric unit with a secondmetric, which indicates where a disturbance of the input data isdirected, a processing unit, which is coupled to the first and secondmetric unit and is designed to generate an optimization problem from thefirst and the second metric, a solution unit, which is coupled to theprocessing unit and is designed to solve the optimization problem bymeans of at least one solution algorithm, wherein the solution indicatesa target disturbance of the input data from sensor data, and agenerating unit, which is coupled to the solution unit and is designedto generate disturbed input data from sensor data for a neural networkby means of the target disturbance.

The generator is designed in particular to carry out the methoddescribed in the preceding for generating disturbed input data. Ittherefore has the same benefits as this method.

In some embodiments, the generator also comprises a third metric unitwith a third metric, which indicates what kind of sensor data thedisturbance is applied to. In this case, the processing unit is alsocoupled to the third metric unit and designed to generate theoptimization problem from at least two metrics of the first, the second,and the third metric.

Moreover, the teachings herein relate to a device for generating aparameter set for a neural network for analyzing sensor data of a driverassistance system with a first analysis unit for generating a firstanalysis by means of the neural network on the basis of training datafor an example sensor data set, the generator described in the precedingfor generating disturbed input data for generating disturbed input dataas training data for the example sensor data set, a second analysis unitfor generating a second analysis of the example sensor data set on thebasis of the disturbed input data by means of the neural network, acomparison unit which is coupled to the first and the second analysisunit and which is designed to compare the first and second analysis, anda generating unit which is coupled to the comparison unit and isdesigned to generate an improved parameter set for the neural network onthe basis of the result of the comparison of the first and secondanalysis.

The device for generating a parameter set is designed in particular tocarry out the method described in the preceding for improving aparameter set of a neural network. It therefore has the same benefits asthis method.

Reference will now be made to the drawings in which the various elementsof embodiments will be given numerical designations and in which furtherembodiments will be discussed.

Specific references to components, process steps, and other elements arenot intended to be limiting. Further, it is understood that like partsbear the same or similar reference numerals when referring to alternateFIGS. It is further noted that the FIGS. are schematic and provided forguidance to the skilled reader and are not necessarily drawn to scale.Rather, the various drawing scales, aspect ratios, and numbers ofcomponents shown in the FIGS. may be purposely distorted to make certainfeatures or relationships easier to understand.

In the exemplary embodiments, sensor data is analyzed by a neuralnetwork or disturbed input data for a neural network is generated fromsuch sensor data. The sensor data in the exemplary embodiments is rawdata from sensors in a vehicle. The sensor may be a camera, a radarsensor, a lidar sensor, or any other sensor which generates sensor datawhich is processed further in a driver assistance system. In thefollowing, it is assumed as an example that the sensor data is digitalimages which have been taken by a camera of a vehicle. The invention,however, may also be applied to other sensor data in the same manner.

With reference to FIG. 1, an exemplary embodiment of the generator 10for generating disturbed input data for a neural network for analyzingdigital images of a driver assistance system is first described.

The generator 10 comprises a first metric unit 1, a second metric unit2, and a third metric unit 3. The first metric unit 1 comprises a firstmetric, which indicates how the magnitude of a change in digital imagesis measured. It is defined by the first metric unit 1 how the magnitudeof a change in digital images is measured. The definition of the firstmetric may be input into the first metric unit 1. However, via aninterface, the first metric unit 1 may also access a database 16 inwhich data with a plurality of possible definitions for metrics thatmeasure the magnitude of a change in digital images is stored. Forexample, the first metric may compare the image distances of two digitalimages and output a value for this image distance. The image distancemay be defined, for example, by the sum of the differences of all pixelvalues of the digital images to be compared.

In the exemplary embodiment, the first metric unit 1 selects adisturbance that is as natural as possible from the database 16. Anatural disturbance is understood to mean a disturbance which influencesdigital images of the surroundings of a vehicle in the same way as mayalso occur due to naturally occurring phenomena in the surroundings ofthe vehicle. The change in a digital image due to a natural disturbancecorresponds, for example, to the change in a digital image as resultsupon the occurrence of weather phenomena, such as upon the occurrence offog, snowfall, or rain. Furthermore, natural disturbances are understoodto mean image changes in which objects are added to the image ordisappear from the image as may also occur in the surroundings of avehicle. For example, in the vicinity of the vehicle, a poster or asticker may be added to an object. Other, non-naturally occurringdisturbances, as may also be contained in the database 16, are not takeninto account by the second metric unit 2, because they are of littlerelevance for the check of a neural network that is used in a driverassistance system.

The second metric unit 2 comprises a second metric, which indicateswhere a disturbance of the input data of the digital images is directed,i.e., the second metric defines where a disturbance of a digital imageis directed. The definition of the second metric may be transferred byan input to the second metric unit 2. In the same way, the second metricunit 2 may also be coupled to the database 16 in which data regarding aplurality of disturbances is stored which is directed at a specificchange in digital images. This may be collections of such disturbances.

In the exemplary embodiment, the second metric unit 2 selects adisturbance that is as plausible as possible from the database 16. Aplausible disturbance is understood to mean a disturbance which resultsin seemingly realistic model output but differs from it in relevantdetails. In the case of a plausible disturbance, a correct segmentation,for example, takes place, in which, however, the lane markings have beenconsistently shifted. Other, non-plausible disturbances, as may also becontained in the database 16, are not taken into account by the secondmetric unit 2, because they are of little relevance for the check of aneural network that is used in a driver assistance system. Severelyimplausible model outputs may namely be detected easily.

The second metric may be directed, for example, at enlarging the size ofall objects that are assigned to a specific class, for example the classof pedestrians. The disturbance thus generates a digital image in whichan object in the starting image that is classified as a pedestrian isenlarged iteratively in all four directions, wherein the resultingsegmentation of the disturbed digital image is combined again. Theresult is a digital image in which all objects that do not belong to theclass of pedestrians remained unchanged, but the objects that belong tothe class of pedestrians are shown enlarged. The other objects are onlychanged insofar as they have been changed by the enlargement of theobjects of the class of pedestrians.

The third metric unit 3 comprises the third metric, which indicates whatkind of digital images the disturbance is applied to. For example, itmay be defined by the metric that the disturbance is only applied todigital images that show other traffic participants, i.e., for example,pedestrians, cyclists, and other vehicles.

The three metric units 1 to 3 are connected to a processing unit 4. Theprocessing unit 4 is designed to generate an optimization problem fromthe three metrics of the first through third metric units 1 to 3. Theoptimization problem comprises, for example, a loss function for aneural network, which contains as a parameter a disturbance parameterand an image resulting from the disturbance (second metric). In the caseof the optimization problem, the minimum of the disturbance parametershould be found, for the digital images that are defined according tothe third metric and under the condition that the magnitude of thechange of the generated image relative to the starting image accordingto the first metric is below a specific value.

The processing unit 4 transfers the optimization problem as a data setto a solution unit 5. The solution unit 5 is coupled to a database 6 inwhich at least one solution algorithm, for example a plurality ofsolution algorithms, for optimization problems are stored. Such solutionalgorithms are generally known. For example, Monte Carlo methods,genetic algorithms, and/or gradient-based methods may be stored in thedatabase 6 and may be accessed by the solution unit 5. By means of thesesolution algorithms, the solution unit 5 may generate a targetdisturbance of the input data of digital images as the solution to theoptimization problem. The target disturbance thus generates a disturbeddigital image, which may be used as input data for a neural network foranalyzing digital images. The neural network is in particular configuredto analyze digital images of a driver assistance system.

The solution unit 5 transfers the target disturbance to a generatingunit 7. The generating unit 7 is also coupled to a database 8, in whicha plurality of digital images are stored.

By means of the target disturbance, the generating unit 7 may disturbdigital images of the database 8 such that disturbed input data 9 of thedigital images for a neural network is generated. The disturbed inputdata 9 is then output from the generating unit 7. With this disturbedinput data 9, a neural network may then be tested, trained, or theparameter set of the neural network may be improved.

An exemplary embodiment of the method for generating disturbed inputdata 9 will be explained with reference to FIG. 2:

In a step S1, a first metric is defined, which indicates how themagnitude of a change in digital images is measured. The first metric,or a data set that describes the first metric, is stored in the firstmetric unit 1.

In a step S2, a second metric is defined, which indicates where adisturbance of the digital images is directed. This second metric, or adata set that describes the second metric, is also stored in the secondmetric unit 2.

Finally, in a step S3, the third metric is defined, which indicates whatkind of digital images a disturbance is applied to. This third metric,or a data set that describes the first metric, is stored in the thirdmetric unit 3.

In a step S4, the data sets that describe the three metrics aretransferred to the processing unit 4.

In a step S5, the processing unit 4 generates an optimization problemfrom a combination of the three metrics. In a step S6, the processingunit 4 transfers a data set that describes the generated optimizationproblem to the solution unit 5.

In a step S7, the solution unit 5 solves the optimization problem bymeans of at least one solution algorithm, which has been transferred tothe solution unit 5, for example, through access to the database 6. Thesolution is a target disturbance for digital images.

In a step S8, a data set for this target disturbance is transferred tothe generating unit 7.

In a step S9, the generating unit 7 generates disturbed digital imagesas input data 9 for a neural network through access to the database 8.This disturbed input data 9 is output in a step S10.

In the following, the method will be explained in detail with referenceto FIGS. 6A to 6C using an example in which objects of the class ofpedestrians are enlarged:

A model M is given. For this model, there is the input x. This input xis a digital image, as it is shown in FIG. 6A. Moreover, the outputM(x)=y is defined. A disturbance is described with A, so that x′=x+Aresults as changed input. The changed output is then y′=M(x+Δ). Thetarget output is described with y″.

In FIG. 6B, the output y of the model M is shown. The digital image xwas segmented, i.e., the pixels of the digital image x were, as shown inFIG. 6B, assigned classes. The following class assignments have therebyresulted:

K1: Sky;

K2: Nature;

K3: Building;

K4: Pedestrian;

K5: Traffic sign;

K6: Road;

K7: Marking.

The target output y″ that should be generated by the disturbance A isshown in FIG. 6C. The target of the disturbance A is to shown thepedestrian enlarged. It is defined as a target disturbance that at mosta shift of individual pixel values by the value of 3 may take place. Thetarget data consists of a concrete image x.

The first metric is then defined as follows:

${d_{1}(\Delta)} = {{\Delta }_{\infty}\max\limits_{pixel}{❘{\Delta({pixel})}❘}}$

The size of the disturbance is thus measured as the maximum pixel valuebetween 0 and 255 in the disturbance A.

The second metric is defined as follows:

$\begin{matrix}{{d_{2}(\Delta)} = {{{{M\left( {x + \Delta} \right)} - {y^{''}_{1}}} = {{y^{\prime} - y^{''}}}_{1}}}} \\{= {\sum\limits_{pixel}{❘{{y^{\prime}({pixel})} - {y^{''}({pixel})}}❘}}}\end{matrix}$

It defines the sum of the pixel deviations from the target output.

The third metric is defined as follows:

${d_{3}\left( x^{\prime} \right)} = \left\{ \begin{matrix}{0;{x^{\prime} = x}} \\{1;{x^{\prime} \neq x}}\end{matrix} \right.$

Thus, according to this third metric, only the input image x has a smallsize. Consequently, the attack only relates to the input image x whend₃(x′)<1 is demanded. The focus with regard to the data to be attackedchanges dramatically when d₃(x′)<2 is demanded: then the attack relatesto all images.

From these three metrics, the optimization problem is then formed asfollows:

$\overset{\_}{\Delta} = {\underset{d_{1} < {3_{1}{d_{3}(x^{\prime})}} < 1}{argmin}\left( {d_{2}(\Delta)} \right)}$

According to the optimization problem, a Δ should be found so that d₂(Δ)is minimal, wherein d₁(Δ)<3 is at x.

This optimization problem may be solved using generally known solutionalgorithms. Because of this, a new adversarial disturbance is obtainedfrom already known (d₁, d₃) and novel (d₂) metrics. New adversarialdisturbances also arise by recombining already known metrics (d₁, . . ., d₃) in a novel manner or their connection with another solutionalgorithm. The method thus allows a design of effectively any number ofnovel adversarial disturbances in a simple manner.

According to a variant of this example, in the first metric, only pixelchanges in an image region may be permitted that are classified as“tree.” The following then results as an optimization problem: A Δshould be found in image regions “tree” in the digital image x so thatd₂(Δ) is minimal, wherein d₁(Δ)<3.

According to another variation of this example, for the third metric, adisturbance may be searched for all images, wherein the first metric d₁and the second metric d₂ are left unchanged. The optimization problemmay then be formulated as follows: A Δ should be found so that d₂(Δ) isminimal for all images, wherein d₁(Δ)<3. In other words, a Δ withd₁(Δ)<3 should be found so that the model output for all input images xlooks like y″.

In the following, another exemplary embodiment of the generator 10 andthe method for generating disturbed input data 9 will be explained:

The generator 10 of the further exemplary embodiment comprises, as inthe first exemplary embodiment, a first metric unit 1 and a secondmetric unit 2. In this case, however, the first metric unit 1 comprisesa first quantity with a plurality of first metrics, which eachdifferently indicate how the magnitude of a change in sensor data ismeasured. In this case, the second metric unit 2 comprises a secondquantity with a plurality of second metrics, which each differentlyindicate where a disturbance of input data 9 from sensor data isdirected. The processing unit 4 coupled to the first 1 and the second 2metric unit is designed in this case to generate the optimizationproblem from any combination of a first metric of the first quantity anda second metric of the second quantity.

The solution unit 5 coupled to the processing unit 4 is then designed tosolve the optimization problem by means of at least one solutionalgorithm, wherein the solution indicates a target disturbance of theinput data 9 from sensor data. Analogously to the first exemplaryembodiment, the generating unit 7 is also designed to generate disturbedinput data 9 from sensor data for a neural network 11 by means of thetarget disturbance.

The method of the further exemplary embodiment runs analogously to themethod of the first exemplary embodiment. However, in this case a firstquantity is defined which contains the first metrics, which eachdifferently indicate how the magnitude of a change in sensor data ismeasured. Furthermore, a second quantity is defined which contains thesecond metrics, which each differently indicate where a disturbance ofsensor data is directed. Any combination of a first metric of the firstquantity and a second metric of the second quantity is then chosen andthe optimization problem is generated from the chosen combination of thefirst and second metric. This is then, as in the method of the firstexemplary embodiment, solved by means of at least one solutionalgorithm, wherein the solution indicates a target disturbance of theinput data 9. By means of the target disturbance, disturbed input data 9is then generated from sensor data for the neural network 11.

With reference to FIG. 3, an exemplary embodiment of a device forgenerating a parameter set for a neural network is described:

The device comprises the database 8 with digital images. The generator10 described with reference to FIG. 1 is connected to this database 8. Aneural network 11 is coupled to the database 8 and the generator 10. Theoutput of the neural network 11 is coupled to a first analysis unit 12and a second analysis unit 13. The first analysis unit 12 generates afirst analysis by means of the neural network 11 on the basis of digitalimages which are fed to the neural network 11 as input data from thedatabase 8. The second analysis unit 13 generates a second analysis onthe basis of disturbed input data 9 that are fed to the neural network11 from the generator 10. To generate the disturbed input data 9, thegenerator 10 accesses the database 8 with the digital images and appliesthe target disturbance generated by the generator 10 to these images.

The first analysis unit 12 and the second analysis unit 13 are coupledto a comparison unit 14. This is designed to compare the first and thesecond analysis with each other.

The comparison unit 14 is coupled to a parameter set generating unit 15.The parameter set generating unit 15 is designed to generate an improvedparameter set for the neural network 11 on the basis of the result ofthe comparison of the first and the second analysis, which wastransferred by the comparison unit 14. The parameter set for the neuralnetwork 11 is generated by the parameter set generating unit 15 so thatthe disturbed input data 9, generated by the generator 10, of thedigital images have a low influence on the analysis of this input databy means of the neural network 11. In particular, the improved parameterset is generated so that the effects of the disturbed input data 9 onthe semantic segmentation of the digital image by means of the neuralnetwork 11 for the disturbed input data does not lead to objects thatare relevant for the safety of a driver assistance system being falselyclassified, these objects disappearing or being shown in a changedmanner. The neural network 11 may thus be trained by means of thedisturbed input data 9 which are generated by the generator 10.

With reference to FIG. 4, an exemplary embodiment of the method forchecking the robustness of a neural network is described:

In a step R1, a neural network with an associated parameter set isprovided. This neural network should be checked.

In a step R2, training data is generated by means of a plurality ofdigital images.

In a step R3, the neural network is trained with training data in agenerally known manner and a first analysis of the digital images isgenerated on the basis of the training data by mean of the neuralnetwork.

In a step R4, disturbed input data is generated as training data for thedigital images by means of the method, as it has been explained withreference to FIG. 2.

In a step R5, a second analysis of the digital images is generated onthe basis of the disturbed input, meaning on the basis of the digitalimages to which the target disturbance has been applied, by means of theneural network.

In a step R6, the first and the second analysis are compared with eachother.

In a step R7, a robustness value depending on the result of thecomparison of the first and second analysis is finally ascertained. Therobustness value is high of the deviation if the second analysis fromthe first analysis is low, in particular with regard to deviations thatare relevant, in particular critical for safety, for the operation of adriver assistance system.

With reference to FIG. 5, a method for improving a parameter set of aneural network is described:

First, the steps R1 through R6 are performed, as it has been explainedwith reference to FIG. 4. Then, in a step R8, an improved parameter setfor the neural network is generated on the basis of the result of thecomparison of the first and second analysis.

LIST OF REFERENCE NUMERALS

-   1 First metric unit-   2 Second metric unit-   3 Third metric unit-   4 Processing unit-   5 Solution unit-   6 Database with solution algorithms-   7 Generating unit-   8 Database with digital image data-   9 Disturbed input data-   10 Generator-   11 Neural network-   12 First analysis unit-   13 Second analysis unit-   14 Comparison unit-   15 Parameter set generating unit-   16 Database with disturbance

The invention has been described in the preceding using variousexemplary embodiments. Other variations to the disclosed embodiments maybe understood and effected by those skilled in the art in practicing theclaimed invention, from a study of the drawings, the disclosure, and theappended claims. In the claims, the word “comprising” does not excludeother elements or steps, and the indefinite article “a” or “an” does notexclude a plurality. A single processor, module or other unit or devicemay fulfil the functions of several items recited in the claims.

The term “exemplary” used throughout the specification means “serving asan example, instance, or exemplification” and does not mean “preferred”or “having advantages” over other embodiments. The term “in particular”used throughout the specification means “serving as an example,instance, or exemplification”.

The mere fact that certain measures are recited in mutually differentdependent claims or embodiments does not indicate that a combination ofthese measures cannot be used to advantage. Any reference signs in theclaims should not be construed as limiting the scope.

What is claimed is:
 1. A method for generating disturbed input data fora neural network for analyzing sensor data of a driver assistancesystem, which sensor data represent one or more digital images, themethod comprising: using a first metric, which indicates how themagnitude of a change in sensor data is measured; using a second metric,which indicates where a disturbance of sensor data is directed, whereinthe first metric and the second metric indicate change of sensor data;generating an optimization problem from a combination of the firstmetric and the second metric; solving the optimization problem using atleast one solution algorithm, wherein the solution indicates a targetdisturbance of the input data; and generating, using the targetdisturbance, disturbed input data comprising changed digital images, forthe neural network.
 2. The method of claim 1, wherein the second metricis directed at a change in the classification of objects.
 3. The methodof claim 1, wherein the second metric is directed at a disappearance ofobjects.
 4. The method of claim 1, wherein the second metric is directedat a change in an object of a specific class.
 5. The method of claim 1,wherein the disturbances that are described by one or more of the firstand the second metric are naturally occurring disturbances.
 6. Themethod of claim 5, wherein one or more of the first and the secondmetrics are stored in a database; and a data set on a naturallyoccurring disturbance that is measured with one or more of the first andthe second metric is loaded from the database.
 7. The method of claim 1,wherein a third metric is defined, which indicates what kind of sensordata a disturbance is applied to; and the optimization problem isgenerated from a combination of at least two metrics of the first, thesecond, and the third metric.
 8. The method of claim 7, wherein thethird metric relates to all sensor data.
 9. The method of claim 7,wherein the third metric relates only to a subset of the sensor data.10. The method of claim 7, wherein the third metric describes onlysensor data that contain a specific object.
 11. The method of claim 1,wherein the solution algorithm comprises iterative methods using thegradients of the neural network for determining the change directions.12. A method for generating disturbed input data for a neural networkfor analyzing sensor data of a driver assistance system, comprising:using a first quantity which contains first metrics, which eachdifferently indicate how the magnitude of a change in sensor data ismeasured; using a second quantity which contains second metrics, whicheach differently indicate where a disturbance of sensor data isdirected; selecting any combination of a first metric from the firstquantity and a second metric from the second quantity; generating anoptimization problem from the chosen combination of the first and secondmetric; solving the optimization problem using at least one solutionalgorithm, wherein the solution indicates a target disturbance of theinput data; and generating, using the target disturbance, disturbedinput data from sensor data for the neural network.
 13. The method ofclaim 12, wherein the first quantity comprises at least two, inparticular at least five, different first metrics.
 14. The method ofclaim 12, wherein the second quantity comprises at least two, inparticular at least five, different second metrics.
 15. The method ofclaim 12, comprising: using a third metric, which indicates what kind ofsensor data a disturbance is applied to; selecting any combination of afirst metric of the first quantity, a second metric of the secondquantity, and the third metric; and generating an optimization problemfrom the chosen combination of the first, second, and third metric. 16.The method of claim 12, comprising: using a solution algorithm quantitythat contains multiple solution algorithms that each solve theoptimization problem differently in order to generate different targetdisturbances of the input data; and selecting any solution algorithm ofthe solution algorithm quantity in order to generate disturbed inputdata from sensor data for the neural network.
 17. A method for checkingthe robustness of a neural network for analyzing sensor data againstdisturbed input data, comprising: providing a neural network with anassociated parameter set; generating training data an example sensordata set; generating a first analysis of the example sensor data set onthe basis of the training data using the neural network; generatingdisturbed input data as training data for the example sensor data setusing the method of claim 1; generating a second analysis of the examplesensor data set on the basis of the disturbed input data using theneural network; comparing the first and second analysis; and determininga robustness value depending on the result of the comparison of thefirst and second analysis.
 18. A method for improving a parameter set ofa neural network for analyzing sensor data in relation to disturbedinput data, comprising: providing a neural network with an associatedparameter set; generating training data using an example sensor dataset; generating a first analysis of the example sensor data set on thebasis of the training data using the neural network; generatingdisturbed input data as training data for the example sensor data setusing the method of claim 1; generating a second analysis of the examplesensor data set on the basis of the disturbed input data using theneural network; comparing the first and second analysis; and generatingan improved parameter set for the neural network on the basis of theresult of the comparison of the first and second analysis.
 19. Agenerator for generating disturbed input data for a neural network foranalyzing sensor data of a driver assistance system, which sensor datarepresent one or more digital images, the generator comprising: a firstmetric unit with a first metric, which indicates how the magnitude of achange in sensor data is measured, a second metric unit with a secondmetric, which indicates where a disturbance of the input data fromsensor data is directed, wherein the first metric and the second metricindicate change of sensor data; a processing unit, which is coupled tothe first and second metric unit and is designed to generate anoptimization problem from the first and the second metric; a solutionunit, which is coupled to the processing unit and is designed to solvethe optimization problem by means of at least one solution algorithm,wherein the solution indicates a target disturbance of the input datafrom sensor data; and a generating unit, which is coupled to thesolution unit and is designed to, using the target disturbance, generatedisturbed input data comprising changed digital images for a neuralnetwork.
 20. A generator for generating disturbed input data for aneural network for analyzing sensor data of a driver assistance system,with a first metric unit with a first quantity which contains firstmetrics, which each differently indicate how the magnitude of a changein sensor data is measured, a second metric unit with a second quantitywhich contains second metrics, which each differently indicate where adisturbance of the input data from sensor data is directed, a processingunit, which is coupled to the first and the second metric unit and isdesigned to generate an optimization problem from any combination of afirst metric of the first quantity and a second metric of the secondquantity, a solution unit, which is coupled to the processing unit andis designed to solve the optimization problem by means of at least onesolution algorithm, wherein the solution indicates a target disturbanceof the input data from sensor data, and a generating unit, which iscoupled to the solution unit and is designed to generate disturbed inputdata from sensor data for a neural network by means of the targetdisturbance.
 21. The generator of claim 19, comprising a third metricunit with a third metric, which indicates what kind of sensor data thedisturbance is applied to, wherein the processing unit is also coupledto the third metric unit and is designed to generate the optimizationproblem from at least two metrics of the first, the second, and thethird metric.